Methods and systems for computing a hash from a three dimensional data set loaded into a resource

ABSTRACT

Methods and system for computing a hash from a three dimensional data set loaded into a three dimensional resource, these methods and systems first determining that an application, executing in a distributed computing environment, locked a three dimensional resource. A identification of a first three dimensional data set loaded into the three dimensional resource by the application is then made. The first three dimensional data set can have a size that is determined in part by a transaction carried out by the application. The performance of an operation on the three dimensional resource by the application is then detected, a first hash is computed on the loaded first set of three dimensional data using a hash function, and the first hash is stored in a repository. Prior to computing the first hash, a portion of the first three dimensional data set can be eliminated, where the portion of the three dimensional data set results from a repetitious transaction.

This U.S. Patent Application claims priority to U.S. Provisional PatentApplication Ser. No. 61/023,867, filed on Jan. 27, 2008; and claimspriority to U.S. Provisional Patent Application Ser. No. 61/108,538,filed on Oct. 26, 2008. The disclosures of these prior applications areconsidered part of the disclosure of this application and areincorporated herein by reference in each of their entirety.

FIELD OF THE INVENTION

This invention relates generally to remotely providing graphics. Morespecifically, this invention relates to remotely providingthree-dimensional graphics.

BACKGROUND OF THE INVENTION

Remote delivery systems that remotely provide applications or desktopshaving enhanced graphics requirements such as those applications anddesktops that employ three-dimensional graphics; can experienceend-user-experience degradation due to said enhanced graphicsrequirements. For example, when delivering a desktop or application thatuses three-dimensional graphics to an end-user of Citrix's XenApp,XenDesktop or XenServer, the end-user may be provided with an end-userexperience of lesser quality. Mitigating this quality degradation causedby the three-dimensional graphics can in some embodiments beaccomplished by altering the way in which desktops and applications areremotely provided to end-users.

There currently exists various different methods and systems forremotely providing two-dimensional graphics. For example, such methodsand systems typically accomplish the remote presentation of twodimensional graphics by: rendering graphics on one computing machine andtransmitting the rendered graphical content to another computingmachine; determining when rendered graphics are stored on a remotecomputing machine, rendering only those graphics not stored on theremote computing machine and transmitting the newly rendered graphics tothe remote computing machine. The methods and systems for remotelyproviding two-dimensional graphics that currently exist, do not takeinto consideration the enhanced graphics requirements ofthree-dimensional graphics. Such enhanced graphics requirements caninclude the increased computing resources needed to both render andtransmit three-dimensional graphics, the complexity of rendering atwo-dimensional representation of a three-dimensional image, and othersuch requirements.

What is needed are methods and systems for remotely providingthree-dimensional graphics and other such graphics that have enhancedgraphics requirements beyond those needed to render two-dimensionalgraphics. These methods and systems would additionally reduce thequality degradation caused by remotely providing three-dimensionalgraphics because these methods and systems would provide the additionalgraphics requirements needed to successfully remote three-dimensionalgraphics without degrading the quality of the remotely providedapplication or desktop.

SUMMARY OF THE INVENTION

In its broadest interpretation, this disclosure describes methods andsystems for remoting three-dimensional graphics. Remoting twodimensional graphics often does not take into consideration the enhancedgraphics requirements needed by three-dimensional images and thereforemay not adequately remote three-dimensional graphics. The presentdisclosure describes methods and systems for remoting three-dimensionalgraphics that interface with three-dimensional graphics technologiessuch that they efficiently handle the multiple graphics componentsassociated with the three-dimensional graphics, that were not associatedwith two dimensional graphics.

In one aspect, a method for remoting three-dimensional graphics includesintercepting, by an agent, a plurality of graphics commands generated bya three-dimensional application executing on a local computing machine.The method includes encoding, by the agent, a first portion of theplurality of graphics commands with a first codec. The method includesencoding, by the agent, a second portion of the plurality of graphicscommands with a second codec. The method includes creating, by theagent, a frame comprising the first portion and the second portion ofthe plurality of graphics commands. The method includes compressing theframe by the agent. The method includes transmitting, by the agent, theframe to a remote computing machine.

In one embodiment, the method includes encoding a first portion of theplurality of graphics commands comprising one of vertex graphicscommands, index graphics commands, and texture graphics commands. Inanother embodiment, the method includes encoding a second portion of theplurality of graphics commands comprising one of vertex graphicscommands, index graphics commands, and texture graphics commands. Instill another embodiment, the method includes encoding a second portionof the plurality of graphics commands different from the first portionof the plurality of graphics commands. In yet another embodiment, themethod includes encoding a second portion of the plurality of graphicscommands substantially similar to the first portion of the plurality ofgraphics commands.

In one embodiment, the method includes intercepting the graphicscommands before they are transmitted to a three-dimensional graphicsdriver. In another embodiment, the method includes redirecting thegraphics commands to a proxy three-dimensional graphics driver. In stillanother embodiment, the method includes creating a frame furthercomprises creating a frame comprising graphics primitives. In still evenanother embodiment, the method includes creating a frame furthercomprises creating a frame comprising DirectX graphics primitives. Instill even another embodiment, the method includes creating a framefurther comprises creating a frame comprising OpenGL graphicsprimitives. In still another embodiment, the method includes creating aframe further comprises creating a frame comprising Direct3D graphicsprimitives.

In another aspect, a system for remoting three-dimensional graphicsincludes a local computing machine, a remote computing machine, and anagent executing on the local computing machine. The local computingmachine executes a three-dimensional application that generates aplurality graphics commands. The remote computing machine is incommunication with the local computing machine via a remote presentationprotocol. The agent intercepts the plurality of graphics commands. Theagent encodes a first portion of the plurality of graphics commands witha first codec. The agent encodes a second portion of the plurality ofgraphics commands with a second codec. The agent creates a framecomprising the first portion and the second portion of the plurality ofgraphics commands. The agent compresses the frame. The agent transmitsthe frame to the remote computing machine.

Taken another way in its broadest interpretation, this disclosuredescribes methods and systems for rendering three-dimensional graphicaldata. Graphics command remoting of three-dimensional graphics can betaxing on a local computing device's central processing unit and onnetwork resources. Similarly, bitmap remoting of three-dimensionalgraphics can also be taxing on a local computing device's centralprocessing unit and on network resources. The present disclosuredescribes methods and systems for rendering three-dimensionalapplications either locally or remotely on a transactional orapplication basis. The hybrid rendering scheme carried out by thepresent disclosure can alleviate the stress placed on a system bycommand and bitmap remoting by spreading out the rendering requests overa local and remote computing machine.

In one aspect, a method for rendering three-dimensional graphical dataincludes intercepting a three-dimensional graphics stream comprisingthree-dimensional graphics commands generated by an applicationexecuting on a first computing machine. The method includes analyzingcharacteristics of a remoting system to determine a location forrendering three-dimensional data from the three-dimensional graphicscommands, the remoting system comprising at least the first computingmachine having a graphics rendering component, a second computingmachine having a graphics rendering component and a network. The methodincludes determining a rendering location based on the remoting systemanalysis. The method includes inducing the application to reinitialize acontext for determining where to render three-dimensional data. Themethod includes rendering three-dimensional data from thethree-dimensional graphics commands at the rendering location,responsive to the inducement.

In one embodiment, the method includes inducing a reinitialization ofthe context by reporting a device state, associated with a device, tothe application. In another embodiment, the method includes reporting adevice state indicating that the device is removed. In still anotherembodiment, the method includes analyzing characteristics associatedwith any of an available bandwidth on the network, a load placed on thefirst computing machine graphics rendering component, a load placed onthe second computing machine graphics rendering component, theapplication, and the three-dimensional graphics commands. In still evenanother embodiment, the method includes analyzing characteristicsassociated with any of an available bandwidth on the network, a loadplaced on the first computing machine graphics rendering component, aload placed on the second computing machine graphics renderingcomponent, the application, and the three-dimensional graphics commands.In yet another embodiment the method includes analyzing characteristicsassociated with any of an available bandwidth on the network, a loadplaced on the first computing machine graphics rendering component, aload placed on the second computing machine graphics renderingcomponent, the application, and the three-dimensional graphics commands.

In one embodiment, the method includes intercepting a three-dimensionalgraphics stream comprising three-dimensional graphics commands andrendered graphics. In another embodiment, the method includesdetermining the rendering location for rendering the three-dimensionaldata is the first computing machine. In still another embodiment, themethod includes inducing the application to reinitialize the context tospecify the first computing machine as the rendering location;rendering, on the first computing machine, the three-dimensional data;and transmitting to the second computing machine the renderedthree-dimensional data. In yet another embodiment, the method includesdetermining the rendering location for rendering the three-dimensionaldata is the second computing machine.

In one embodiment, the method includes inducing the application toreinitialize the context to specify the second computing machine as therendering location; transmitting the three-dimensional graphics streamto the second computing machine; and rendering, on the second computingmachine, the three-dimensional data. In another embodiment, the methodincludes determining to render a first portion of the three-dimensionaldata on the first computing machine; inducing the application toreinitialize the context to specify the first computing machine as therendering location for the first portion of the three-dimensional data;rendering the first portion of three-dimensional data on the firstcomputing machine; and storing, on the first computing machine, therendered first portion of three-dimensional data as shared resources. Instill another embodiment, the method includes intercepting thethree-dimensional graphics stream comprising three-dimensional graphicscommands and the shared resources; determining to render thethree-dimensional data on the second computing machine; inducing theapplication to reinitialize the context to specify the second computingmachine as the rendering location; transmitting the three-dimensionalgraphics stream to the second computing machine; and rendering, on thesecond computing machine, the three-dimensional data from thethree-dimensional graphics commands.

In another aspect, a system for rendering three-dimensional graphicaldata includes a first computing machine, a second computing machine, aremoting system, and a rendering agent. The first computing machineexecutes an application that generates three-dimensional graphicscommands. The first computing machine has a graphics renderingcomponent. The second computing machine, in communication with the firstcomputing machine via a network, has a graphics rendering component. Theremoting system includes the first computing machine, the secondcomputing machine and the network. The rendering agent executes on thefirst computing machine and intercepts a three-dimensional graphicsstream comprising the three-dimensional graphics commands. The renderingagent analyzes characteristics of the remoting system to determine alocation for rendering three-dimensional data from the three-dimensionalgraphics commands. The rendering agent determines a rendering locationbased on the remoting system analysis. The rendering agent induces theapplication to reinitialize a context for determining where to renderthree-dimensional data. In one embodiment, the rendering agent includesmeans for inducing the reinitialization of the context by reporting adevice state, associated with a device, to the application. In anotherembodiment, the rendering agent includes means for reporting a devicestate that indicates the device is removed.

In another broad interpretation, this disclosure describes methods andsystems for computing a hash from a three-dimensional data set loadedinto a three-dimensional resource where the three-dimensional data sethas a variable size. In most applications little thought is given as towhether graphical processing unit resources are conservatively used byapplications. For example, many applications load resources directlyinto a driver during the production of each frame. Many of these framesare synthetically generated by a looping graphics engine in theapplication, therefore there can be quite a bit of redundancy withregard to the graphics content of co-joining frames or frames that areonly a few frames apart. The present disclosure describes methods andsystems for mitigating such redundancy by advantageously using thetransactional nature of loading resources into a three-dimensionalresource to compute a hash on data generated during the transaction. Bycaching based on a data set created during a transaction, rather than abyte stream, repetitious data resulting from repetitious transactionscan be thrown out prior to compression. What results is a efficientcompression of the three-dimensional data loaded into the resource freeof repetitions.

In one aspect, described is a method for computing a hash from athree-dimensional data set loaded into a three-dimensional resource, thethree-dimensional data set having a variable size. A determination isfirst made as to whether an application, executing in a distributedcomputing environment, locked a three-dimensional resource. Anidentification is then made of a first three-dimensional data set loadedinto the three-dimensional resource by the application, the firstthree-dimensional data set having a size determined in part by atransaction carried out by the application. The method further comprisesdetecting the application performed an operation on thethree-dimensional resource, computing a first hash on the loaded firstset of three-dimensional data, the first hash computed using a hashfunction, and storing the first hash in a repository.

In one embodiment, the method further comprises eliminating a portion ofthe first three-dimensional data set responsive to detecting theapplication performed an operation, the portion of the three-dimensionaldata set resulting from a repetitious transaction.

Other embodiments include determining the application locked athree-dimensional resource used by the application. While still otherembodiments include computing the has function using an adler-32 hashfunction.

In one embodiment, the three-dimensional resource is one of either of avertex, index, shade, texture, and surface. Other embodiments includeidentifying the first three-dimensional data set further comprisesidentifying a first three-dimensional data set having a size determinedin part by a transaction carried out by the application, the transactioncomprising at least copying three-dimensional data into thethree-dimensional resource. In such an embodiment, the transactionfurther comprises an end event comprising at least unlocking thethree-dimensional resource. Still other embodiments include identifyingthe second three-dimensional data set further comprises identifying asecond three-dimensional data set having a size determined in part by atransaction carried out by the application, the transaction comprisingat least copying three-dimensional data into the three-dimensionalresource. In such embodiments, the transaction further comprises an endevent comprising at least unlocking the three-dimensional resource.

In still another embodiment, the first three-dimensional data set has asize determined in part by a transaction carried out by the application,the size representative in part of a period of time spanning from astart event to an end event. The start event, in one embodiment, furthercomprises locking the three-dimensional resource. The end event, in oneembodiment, further comprises unlocking the three-dimensional resource.

In one embodiment, the first hash is a cache.

Still other embodiments include a method further comprising identifyinga second three-dimensional data set loaded into the three-dimensionalresource by the application, the second three-dimensional data sethaving a size different from the size of the first three-dimensionaldata set. The method also includes detecting the application performedan operation on the three-dimensional resource, computing a second hashon the loaded second set of three-dimensional data, the second hashcomputed using the hash function, and storing the second hash in arepository. In another embodiment, the second hash is a cache. Stillanother embodiment includes a method further comprising identifying thesecond three-dimensional data set further comprises identifying a secondthree-dimensional data set having a size determined in part by atransaction carried out by the application, the size representative inpart of a period of time spanning from a start event to an end event.The start event can, in some embodiments, comprise locking thethree-dimensional resource, while the end event can, in someembodiments, comprise unlocking the three-dimensional resource.

In yet another broad interpretation, this disclosure describes methodsand systems for detecting dirty regions within a frame encompassingthree-dimensional graphics and three-dimensional graphics primitives. Aparticular three-dimensional graphic is, in many cases, transmitted froma local computing machine to a remote computing machine once the bitmaprendered and saved to a back buffer is displayed by flipping thecontents of the back buffer so that a front buffer can access therendered image and display it on the screen. To remotely provide athree-dimensional graphic, one would wait until the renderedthree-dimensional graphic is displayed on the screen and then transmitthe new bitmap to the remote computing machine. This can be a resourceintensive process that requires a lot of bandwidth. The presentdisclosure describes methods and systems that can review the renderedthree-dimensional graphics before they are flipped to the front bufferfor display on the screen. Reviewing the rendered image before it isflipped to the front buffer allows for a determination to be made as towhat sections of the bitmap were changed. This further allows for theextraction of changed or dirty regions such that only those regions aretransmitted to the remote computing machine. Transmitting only sectionsof the rendered three-dimensional graphic reduces the amount ofbandwidth needed to transmit the graphic over a network.

In one aspect, described is an embodiment of a method for detecting adirty region within a frame encompassing three-dimensional graphics andthree-dimensional graphics primitives. An agent executing on a localcomputing machine, intercepts a function call issued by an applicationgenerating three-dimensional graphics. The agent further identifies,responsive to intercepting the function call, at least one locationwithin a frame containing three-dimensional graphics commands generatedby the application, the location corresponding to a drawing region. Theagent further determines at least a portion of the three-dimensionalgraphics commands, associated with the at least one location within theframe, changed. Further the agent extracts the changed portion of thethree-dimensional graphics commands.

In one embodiment, the function call comprises a call to set a drawingregion, where a drawing region can in some embodiments be a drawingregion on a display device.

Other embodiments include a method where the three-dimensional graphicaldata is rendered to a buffer. The function call, in some embodiments,can include a call to display contents of the buffer.

In another embodiment, the frame comprises a three-dimensional graphicslibrary, while in other embodiments the at least one location furthercomprises identifying at least one viewport.

Some embodiments include an agent that further identifies a plurality oflocations within a frame, each location containing three-dimensionalgraphics commands generated by the application, the locationscorresponding to a drawing regions. After identifying the plurality oflocations, the agent determines portions of the three-dimensionalgraphics commands, associated with the plurality of locations within theframe, changed. The agent further extracts the changed portions of thethree-dimensional graphics commands from the plurality of drawingregion, and transmits the changed portions to a remote computing machinecommunicating with the local computing machine.

In yet another embodiment the method further comprises transmitting thechanged portion to a remote computing machine communicating with thelocal computing machine.

Other aspects describe a system for detecting a dirty region within aframe encompassing three-dimensional graphics and three-dimensionalgraphics primitives that includes a local computing machine executing anapplication generating three-dimensional graphics, and issuing at leastone function call. The system further comprises an agent executing onthe local computing machine to: intercept the issued at least onefunction call; identify, responsive to intercepting the function call,at least one location within a frame containing three-dimensionalgraphics commands generated by the application, the locationcorresponding to a drawing region; determine at least a portion of thethree-dimensional graphics commands, associated with the at least onelocation within the frame, changed; and extract the changed portion ofthe three-dimensional graphics commands.

Sill other aspects describe a computer readable medium havinginstructions executable by a processor to detect a dirty region within aframe encompassing three-dimensional graphics and three-dimensionalgraphics primitives. The computer readable medium further comprisinginstructions to intercept, via an agent executing on a local computingmachine, a function call issued by an application generatingthree-dimensional graphics. Also included are instructions to identify,by the agent and responsive to intercepting the function call, at leastone location within a frame containing three-dimensional graphicscommands generated by the application, the location corresponding to adrawing region. The computer readable medium further includesinstructions to determine at least a portion of the three-dimensionalgraphics commands, associated with the at least one location within theframe, changed; and instructions to extract the changed portion of thethree-dimensional graphics commands.

In still yet another broad interpretation, this disclosure describesmethods and systems for improving resource utilization when remotingthree-dimensional graphics commands generated by an applicationexecuting on a local computing machine. In some cases, remoting a twodimensional or three-dimensional application requires the rendering ofall frames as received from the application and at a standard speed,then transmitting the rendered frames to a remote computing machine.Systems and methods that utilize this process may find that the networkinstalled between the remote computing machine and local computingmachine is insufficient to transmit data at the same speed as the speedat which the data is rendered. The present disclosure describes methodsand systems that can prevent the rendering of three-dimensional graphicsuntil the previous frame is received by the remote computing device.Thus, the local computing device only renders frames once previousframes have been received by the remote computing device.

In one aspect, described herein is an embodiment of a method forimproving resource utilization when remoting three-dimensional graphicscommands generated by an application executing on a local computingmachine. This method can include intercepting, by an agent executing ona local computing machine, a first call to a drawing library, the firstcall generated by an application requesting information from the drawinglibrary. The agent then transmits the first frame, generated by theapplication, to a remote computing machine, and further prevents, uponfailing to receive a notification indicating the first frame wasrendered, a return of the first call to the requesting application. Oncethe agent receives a notification generated by the remote computingmachine and indicating the first frame was rendered, the agent permitsthe return of the first call to the requesting application.

In one embodiment, the first call is a present call requesting toexchange a currently displayed image with an image rendered from thefirst frame. Still other embodiments include preventing the return ofthe first call for a predetermined period of time. The first frame, insome embodiments, comprises three-dimensional graphics commandsgenerated by the application.

Other embodiments include a drawing library that is a DirectX library,while other embodiments include a drawing library that is an OpenGLlibrary.

Some embodiments include receiving a notification that a notificationthat no further data can be transmitted over a network used by the localcomputing machine to communicate with the remote computing machine, anddetermining based on that information that the network is busy.

Other embodiments include a method further comprising transmitting, bythe agent, a second frame generated by the application. The agentreceives a notification indicating the second frame was rendered, andintercepts a second call generated by the application requestinginformation from the drawing library. The agent then identifies anindicator that the second frame was rendered, and permits the secondcall to return to the requesting application.

In other aspects, a system for improving resource utilization whenremoting three-dimensional graphics commands generated by an applicationexecuting on a local computing machine. The system includes a localcomputing machine that executes an application which generates a firstframe, and generates a first call to a drawing library to requestinformation from the drawing library. Also included in an agentexecuting on the local computing machine to do the following: interceptthe first call to the drawing library; transmit the first frame to aremote computing machine; prevent, upon failing to receive notificationindicating the first frame was rendered, a return of the first call tothe requesting application; receive a notification, generated by theremote computing machine, indicating that the first frame was rendered;and permit the return of the first call to the requesting application.

Still other aspects include a computer readable medium havinginstructions executable by a processor to improve resource utilizationwhen remoting three-dimensional graphics commands generated by anapplication executing on a local computing machine. The computerreadable medium includes instructions to intercept, by an agentexecuting on a local computing machine, a first call to a drawinglibrary, the first call generated by an application requestinginformation from the drawing library, and instructions to transmit, bythe agent, a first frame, generated by the application, to a remotecomputing machine. Also included are instructions to prevent, uponfailing to receive a notification indicating the first frame wasrendered, a return of the first call to the requesting application;instructions to receive, by the agent, a notification, generated by theremote computing machine, indicating the first frame was rendered; andinstructions to permit, by the agent, the return of the first call tothe requesting application.

DETAILED DESCRIPTION OF THE DRAWINGS

The following figures depict certain illustrative embodiments of methodsand systems for either remotely providing three-dimensional graphics,methods and systems for the hybrid rendering of such graphics, the useof hash based caching to compress three-dimensional graphics, thedetection of dirty regions within a three-dimensional drawing region,and improving resource utilization by delaying the rendering ofthree-dimensional data during remote presentation of three-dimensionalgraphics, where like reference numerals refer to like elements. Eachdepicted embodiment is illustrative of the methods and systems and notlimiting.

FIG. 1A is a block diagram illustrative of an embodiment of aremote-access, networked environment with a client machine thatcommunicates with a server.

FIG. 1B and 1C are block diagrams illustrative of an embodiment ofcomputing machines for practicing the methods and systems describedherein.

FIG. 2 is a block diagram illustrative of an embodiment of a system forremoting three-dimensional graphics.

FIG. 3 is a block diagram illustrative of an embodiment of a system forremoting three-dimensional graphics.

FIG. 4 is a flow diagram illustrative of an embodiment of a method forremotely providing three-dimensional graphics.

FIGS. 5A-5B are flow diagrams illustrative of embodiments of methods forremotely providing three-dimensional graphics to a client machine.

FIGS. 6A-6B are flow diagrams illustrative of embodiments of methods forrendering three-dimensional graphical data.

FIG. 7 is a flow diagram illustrative of an embodiment of a method forcomputing a hash of three-dimensional graphical data.

FIGS. 8A-8B are block diagrams of a screen displaying an embodiment of athree-dimensional image and the viewport set when that three-dimensionalimage is changed.

FIG. 9 is a flow diagram of an embodiment of a method for detectingdirty regions generated by an application outputting three-dimensionalgraphics commands.

FIG. 10 is a flow diagram of an embodiment for improving resourceutilization.

DETAILED DESCRIPTION

FIG. 1A illustrates one embodiment of a computing environment 101 thatincludes one or more client machines 102A-102N in communication withservers 106A-106N, and a network 104 installed in between the clientmachines 102A-102N and the servers 106A-106N. In some embodiments,client machines 102A-10N may be referred to as a single client machine102 or a single group of client machines 102, while servers may bereferred to as a single server 106 or a single group of servers 106. Oneembodiment includes a single client machine 102 communicating with morethan one server 106, another embodiment includes a single server 106communicating with more than one client machine 102, while anotherembodiment includes a single client machine 102 communicating with asingle server 106.

A client machine 102 within the computing environment may in someembodiments, be referenced by any one of the following terms: clientmachine(s) 102; client(s); client computer(s); client device(s); clientcomputing device(s); client node(s); endpoint(s); endpoint node(s); or asecond machine. The server 106 in some embodiments may be referenced byany one of the following terms: server(s), server farm(s), hostcomputing device(s), or a first machine(s).

The client machine 102 can in some embodiments execute, operate orotherwise provide an application that can be any one of the following:software; a program; executable instructions; a web browser; a web-basedclient; a client-server application; a thin-client computing client; anActiveX control; a Java applet; software related to voice over internetprotocol (VoIP) communications like a soft IP telephone; an applicationfor streaming video and/or audio; an application for facilitatingreal-time-data communications; a HTTP client; a FTP client; an Oscarclient; a Telnet client; or any other type and/or form of executableinstructions capable of executing on client machine 102. Still otherembodiments may include a computing environment 101 with an applicationthat is any of either server-based or remote-based, and an applicationthat is executed on the server 106 on behalf of the client machine 102.Further embodiments of the computing environment 101 include a server106 configured to display output graphical data to a client machine 102using a thin-client or remote-display protocol, where the protocol usedcan be any one of the following protocols: the Independent ComputingArchitecture (ICA) protocol manufactured by Citrix Systems, Inc. of Ft.Lauderdale, Fla.; or the Remote Desktop Protocol (RDP) manufactured bythe Microsoft Corporation of Redmond, Wash.

In one embodiment, the client machine 102 can be a virtual machine 102Csuch as those manufactured by XenSolutions, Citrix Systems, IBM, VMware,or any other virtual machine able to implement the methods and systemsdescribed herein.

The computing environment 101 can, in some embodiments, include morethan one server 106A-106N where the servers 106A-106N are: groupedtogether as a single server 106 entity, logically-grouped together in aserver farm 106; geographically dispersed and logically grouped togetherin a server farm 106, located proximate to each other and logicallygrouped together in a server farm 106. Geographically dispersed servers106A-106N within a server farm 106 can, in some embodiments, communicateusing a WAN, MAN, or LAN, where different geographic regions can becharacterized as: different continents; different regions of acontinent; different countries; different states; different cities;different campuses; different rooms; or any combination of the precedinggeographical locations. In some embodiments the server farm 106 may beadministered as a single entity or in other embodiments may includemultiple server farms 106. The computing environment 101 can includemore than one server 106A-106N grouped together in a single server farm106 where the server farm 106 is heterogeneous such that one server106A-106N is configured to operate according to a first type ofoperating system platform (e.g., WINDOWS NT, manufactured by MicrosoftCorp. of Redmond, Wash.), while one or more other servers 106A-106N areconfigured to operate according to a second type of operating systemplatform (e.g., Unix or Linux); more than one server 106A-106N isconfigured to operate according to a first type of operating systemplatform (e.g., WINDOWS NT), while another server 106A-106N isconfigured to operate according to a second type of operating systemplatform (e.g., Unix or Linux); or more than one server 106A-106N isconfigured to operate according to a first type of operating systemplatform (e.g., WINDOWS NT) while more than one of the other servers106A-106N are configured to operate according to a second type ofoperating system platform (e.g., Unix or Linux).

The computing environment 101 can in some embodiments include a server106 or more than one server 106 configured to provide the functionalityof any one of the following server types: a file server; an applicationserver; a web server; a proxy server; an appliance; a network appliance;a gateway; an application gateway; a gateway server; a virtualizationserver; a deployment server; a SSL VPN server; a firewall; a web server;an application server or as a master application server; a server 106configured to operate as an active direction; a server 106 configured tooperate as application acceleration application that provides firewallfunctionality, application functionality, or load balancingfunctionality, or other type of computing machine configured to operateas a server 106. In some embodiments, a server 106 may include a remoteauthentication dial-in user service such that the server 106 is a RADIUSserver. Embodiments of the computing environment 101 where the server106 comprises an appliance, the server 106 can be an appliancemanufactured by any one of the following manufacturers: the CitrixApplication Networking Group; Silver Peak Systems, Inc; RiverbedTechnology, Inc.; F5 Networks, Inc.; or Juniper Networks, Inc. Someembodiments include a server 106 with the following functionality: afirst server 106A that receives requests from a client machine 102,forwards the request to a second server 106B, and responds to therequest generated by the client machine with a response from the secondserver 106B; acquires an enumeration of applications available to theclient machines 102 and address information associated with a server 106hosting an application identified by the enumeration of applications;presents responses to client requests using a web interface;communicates directly with the client 102 to provide the client 102 withaccess to an identified application; receives output data, such asdisplay data, generated by an execution of an identified application onthe server 106.

The server 106 can be configured to execute any one of the followingapplications: an application providing a thin-client computing or aremote display presentation application; any portion of the CITRIXACCESS SUITE by Citrix Systems, Inc. like the METAFRAME or CITRIXPRESENTATION SERVER; MICROSOFT WINDOWS Terminal Services manufactured bythe Microsoft Corporation; or an ICA client, developed by CitrixSystems, Inc. Another embodiment includes a server 106 configured toexecute an application so that the server may function as an applicationserver such as any one of the following application server types: anemail server that provides email services such as MICROSOFT EXCHANGEmanufactured by the Microsoft Corporation; a web or Internet server; adesktop sharing server; or a collaboration server. Still otherembodiments include a server 106 that executes an application that isany one of the following types of hosted servers applications:GOTOMEETING provided by Citrix Online Division, Inc.; WEBEX provided byWebEx, Inc. of Santa Clara, Calif.; or Microsoft Office LIVE MEETINGprovided by Microsoft Corporation.

In one embodiment, the server 106 may be a virtual machine 106B such asthose manufactured by Citrix Systems, IBM, VMware, or any other virtualmachine able to implement the methods and systems described herein.

Client machines 102 may function, in some embodiments, as a client nodeseeking access to resources provided by a server 106, or as a server 106providing other clients 102A-102N with access to hosted resources. Oneembodiment of the computing environment 101 includes a server 106 thatprovides the functionality of a master node. Communication between theclient machine 102 and either a server 106 or servers 106A-106N can beestablished via any of the following methods: direct communicationbetween a client machine 102 and a server 106A-106N in a server farm106; a client machine 102 that uses a program neighborhood applicationto communicate with a server 106 a-106 n in a server farm 106; or aclient machine 102 that uses a network 104 to communicate with a server106A-106N in a server farm 106. One embodiment of the computingenvironment 101 includes a client machine 102 that uses a network 104 torequest that applications hosted by a server 106A-106N in a server farm106 execute, and uses the network 104 to receive from the server106A-106N graphical display output representative of the applicationexecution. In other embodiments, a master node provides thefunctionality required to identify and provide address informationassociated with a server 106 hosting a requested application. Stillother embodiments include a master node that can be any one of thefollowing: a server 106A-106N within the server farm 106; a remotecomputing machine connected to the server farm 106 but not includedwithin the server farm 106; a remote computing machine connected to aclient 102 but not included within a group of client machines 102; or aclient machine 102.

The network 104 between the client machine 102 and the server 106 is aconnection over which data is transferred between the client machine 102and the server 106. Although the illustration in FIG. 1A depicts anetwork 104 connecting the client machines 102 to the servers 106, otherembodiments include a computing environment 101 with client machines 102installed on the same network as the servers 106. Other embodiments caninclude a computing environment 101 with a network 104 that can be anyof the following: a local-area network (LAN); a metropolitan areanetwork (MAN); a wide area network (WAN); a primary network 104comprised of multiple sub-networks 104′ located between the clientmachines 102 and the servers 106; a primary public network 104 with aprivate sub-network 104′; a primary private network 104 with a publicsub-network104′; or a primary private network 104 with a privatesub-network 104′. Still further embodiments include a network 104 thatcan be any of the following network types: a point to point network; abroadcast network; a telecommunications network; a data communicationnetwork; a computer network; an ATM (Asynchronous Transfer Mode)network; a SONET (Synchronous Optical Network) network; a SDH(Synchronous Digital Hierarchy) network; a wireless network; a wirelinenetwork; a network 104 that includes a wireless link where the wirelesslink can be an infrared channel or satellite band; or any other networktype able to transfer data from client machines 102 to servers 106 andvice versa to accomplish the methods and systems described herein.Network topology may differ within different embodiments, possiblenetwork topologies include: a bus network topology; a star networktopology; a ring network topology; a repeater-based network topology; atiered-star network topology; or any other network topology abletransfer data from client machines 102 to servers 106, and vice versa,to accomplish the methods and systems described herein. Additionalembodiments may include a network 104 of mobile telephone networks thatuse a protocol to communicate among mobile devices, where the protocolcan be any one of the following: AMPS; TDMA; CDMA; GSM; GPRS UMTS; orany other protocol able to transmit data among mobile devices toaccomplish the systems and methods described herein.

Illustrated in FIG. 1B is an embodiment of a computing device 100, wherethe client machine 102 and server 106 illustrated in FIG. 1A can bedeployed as and/or executed on any embodiment of the computing device100 illustrated and described herein. Included within the computingdevice 100 is a system bus 150 that communicates with the followingcomponents: a central processing unit 121; a main memory 122; storagememory 128; an input/output (I/O) controller 123; display devices124A-124N; an installation device 116; and a network interface 118. Inone embodiment, the storage memory 128 includes: an operating system,software routines, and a client agent 120. The I/O controller 123, insome embodiments, is further connected to a key board 126, and apointing device 127. Other embodiments may include an I/O controller 123connected to more than one input/output device 130A-130N.

FIG. 1C illustrates one embodiment of a computing device 100, where theclient machine 102 and server 106 illustrated in FIG. 1A can be deployedas and/or executed on any embodiment of the computing device 100illustrated and described herein. Included within the computing device100 is a system bus 150 that communicates with the following components:a bridge 170, and a first I/O device 130A. In another embodiment, thebridge170 is in further communication with the central processing unit121, where the central processing unit 121 can further communicate witha second I/O device 130B, a main memory 122, and a cache memory 140.Included within the central processing unit 121, are I/O ports, a memoryport 103, and a main processor.

Embodiments of the computing machine 100 can include a centralprocessing unit 121 characterized by any one of the following componentconfigurations: logic circuits that respond to and process instructionsfetched from the main memory unit 122; a microprocessor unit, such as:those manufactured by Intel Corporation; those manufactured by MotorolaCorporation; those manufactured by Transmeta Corporation of Santa Clara,Calif.; the RS/6000 processor such as those manufactured byInternational Business Machines; a processor such as those manufacturedby Advanced Micro Devices; or any other combination of logic circuitscapable of executing the systems and methods described herein. Stillother embodiments of the central processing unit 122 may include anycombination of the following: a microprocessor, a microcontroller, acentral processing unit with a single processing core, a centralprocessing unit with two processing cores, or a central processing unitwith more than one processing cores.

One embodiment of the computing machine 100 includes a centralprocessing unit 121 that communicates with cache memory 140 via asecondary bus also known as a backside bus, while another embodiment ofthe computing machine 100 includes a central processing unit 121 thatcommunicates with cache memory via the system bus 150. The local systembus 150 can, in some embodiments, also be used by the central processingunit to communicate with more than one type of I/O devices 130A-130N. Insome embodiments, the local system bus 150 can be any one of thefollowing types of buses: a VESA VL bus; an ISA bus; an EISA bus; aMicroChannel Architecture (MCA) bus; a PCI bus; a PCI-X bus; aPCI-Express bus; or a NuBus. Other embodiments of the computing machine100 include an I/O device 130A-130N that is a video display 124 thatcommunicates with the central processing unit 121 via an AdvancedGraphics Port (AGP). Still other versions of the computing machine 100include a processor 121 connected to an I/O device 130A-130N via any oneof the following connections: HyperTransport, Rapid I/O, or InfiniBand.Further embodiments of the computing machine 100 include a communicationconnection where the processor 121 communicates with one I/O device 130Ausing a local interconnect bus and with a second I/O device 130B using adirect connection.

Included within some embodiments of the computing device 100 is each ofa main memory unit 122 and cache memory 140. The cache memory 140 willin some embodiments be any one of the following types of memory: SRAM;BSRAM; or EDRAM. Other embodiments include cache memory 140 and a mainmemory unit 122 that can be any one of the following types of memory:Static random access memory (SRAM), Burst SRAM or SynchBurst SRAM(BSRAM), Dynamic random access memory (DRAM), Fast Page Mode DRAM (FPMDRAM), Enhanced DRAM (EDRAM), Extended Data Output RAM (EDO RAM),Extended Data Output DRAM (EDO DRAM), Burst Extended Data Output DRAM(BEDO DRAM), Enhanced DRAM (EDRAM), synchronous DRAM (SDRAM), JEDECSRAM, PC100 SDRAM, Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM(ESDRAM), SyncLink DRAM (SLDRAM), Direct Rambus DRAM (DRDRAM),Ferroelectric RAM (FRAM), or any other type of memory device capable ofexecuting the systems and methods described herein. The main memory unit122 and/or the cache memory 140 can in some embodiments include one ormore memory devices capable of storing data and allowing any storagelocation to be directly accessed by the central processing unit 121.Further embodiments include a central processing unit 121 that canaccess the main memory 122 via one of either: a system bus 150; a memoryport 103; or any other connection, bus or port that allows the processor121 to access memory 122.

One embodiment of the computing device 100 provides support for any oneof the following installation devices 116: a floppy disk drive forreceiving floppy disks such as 3.5-inch, 5.25-inch disks or ZIP disks, aCD-ROM drive, a CD-R/RW drive, a DVD-ROM drive, tape drives of variousformats, USB device, a bootable medium, a bootable CD, a bootable CD forGNU/Linux distribution such as KNOPPIX®, a hard-drive or any otherdevice suitable for installing applications or software. Applicationscan in some embodiments include a client agent 120, or any portion of aclient agent 120. The computing device 100 may further include a storagedevice 128 that can be either one or more hard disk drives, or one ormore redundant arrays of independent disks; where the storage device isconfigured to store an operating system, software, programsapplications, or at least a portion of the client agent 120. A furtherembodiment of the computing device 100 includes an installation device116 that is used as the storage device 128.

Furthermore, the computing device 100 may include a network interface118 to interface to a Local Area Network (LAN), Wide Area Network (WAN)or the Internet through a variety of connections including, but notlimited to, standard telephone lines, LAN or WAN links (e.g., 802.11,T1, T3, 56kb, X.25, SNA, DECNET), broadband connections (e.g., ISDN,Frame Relay, ATM, Gigabit Ethernet, Ethernet-over-SONET), wirelessconnections, or some combination of any or all of the above. Connectionscan also be established using a variety of communication protocols(e.g., TCP/IP, IPX, SPX, NetBIOS, Ethernet, ARCNET, SONET, SDH, FiberDistributed Data Interface (FDDI), RS232, RS485, IEEE 802.11, IEEE802.11a, IEEE 802.11b, IEEE 802.11g, CDMA, GSM, WiMax and directasynchronous connections). One version of the computing device 100includes a network interface 118 able to communicate with additionalcomputing devices 100′ via any type and/or form of gateway or tunnelingprotocol such as Secure Socket Layer (SSL) or Transport Layer Security(TLS), or the Citrix Gateway Protocol manufactured by Citrix Systems,Inc. Versions of the network interface 118 can comprise any one of: abuilt-in network adapter; a network interface card; a PCMCIA networkcard; a card bus network adapter; a wireless network adapter; a USBnetwork adapter; a modem; or any other device suitable for interfacingthe computing device 100 to a network capable of communicating andperforming the methods and systems described herein.

Embodiments of the computing device 100 include any one of the followingI/O devices 130A-130N: a keyboard 126; a pointing device 127; mice;trackpads; an optical pen; trackballs; microphones; drawing tablets;video displays; speakers; inkjet printers; laser printers; anddye-sublimation printers; or any other input/output device able toperform the methods and systems described herein. An I/O controller 123may in some embodiments connect to mulitple I/O devices 103A-130N tocontrol the one or more I/O devices. Some embodiments of the I/O devices130A-130N may be configured to provide storage or an installation medium116, while others may provide a universal serial bus (USB) interface forreceiving USB storage devices such as the USB Flash Drive line ofdevices manufactured by Twintech Industry, Inc. Still other embodimentsof an I/O device 130 may be a bridge between the system bus 150 and anexternal communication bus, such as: a USB bus; an Apple Desktop Bus; anRS-232 serial connection; a SCSI bus; a FireWire bus; a FireWire 800bus; an Ethernet bus; an AppleTalk bus; a Gigabit Ethernet bus; anAsynchronous Transfer Mode bus; a HIPPI bus; a Super HIPPI bus; aSerialPlus bus; a SCI/LAMP bus; a FibreChannel bus; or a Serial Attachedsmall computer system interface bus.

In some embodiments, the computing machine 100 can connect to multipledisplay devices 124A-124N, in other embodiments the computing device 100can connect to a single display device 124, while in still otherembodiments the computing device 100 connects to display devices124A-124N that are the same type or form of display, or to displaydevices that are different types or forms. Embodiments of the displaydevices 124A-124N can be supported and enabled by the following: one ormultiple I/O devices 130A-130N; the I/O controller 123; a combination ofI/O device(s) 130A-130N and the I/O controller 123; any combination ofhardware and software able to support a display device 124A-124N; anytype and/or form of video adapter, video card, driver, and/or library tointerface, communicate, connect or otherwise use the display devices 124a-124 n. The computing device 100 may in some embodiments be configuredto use one or multiple display devices 124A-124N, these configurationsinclude: having multiple connectors to interface to multiple displaydevices 124 a-124 n; having multiple video adapters, with each videoadapter connected to one or more of the display devices 124A-124N;having an operating system configured to support multiple displays124A-124N; using circuits and software included within the computingdevice 100 to connect to and use multiple display devices 124A-124N; andexecuting software on the main computing device 100 and multiplesecondary computing devices to enable the main computing device 100 touse a secondary computing device's display as a display device 124A-124Nfor the main computing device 100. Still other embodiments of thecomputing device 100 may include multiple display devices 124A-124Nprovided by multiple secondary computing devices and connected to themain computing device 100 via a network.

In some embodiments of the computing machine 100, an operating systemmay be included to control task scheduling and access to systemresources. Embodiments of the computing device 100 can run any one ofthe following operation systems: versions of the MICROSOFT WINDOWSoperating systems such as WINDOWS 3.x; WINDOWS 95; WINDOWS 98; WINDOWS2000; WINDOWS NT 3.51; WINDOWS NT 4.0; WINDOWS CE; WINDOWS XP; andWINDOWS VISTA; the different releases of the Unix and Linux operatingsystems; any version of the MAC OS manufactured by Apple Computer; OS/2,manufactured by International Business Machines; any embedded operatingsystem; any real-time operating system; any open source operatingsystem; any proprietary operating system; any operating systems formobile computing devices; or any other operating system capable ofrunning on the computing device and performing the operations describedherein. One embodiment of the computing machine 100 has multipleoperating systems installed thereon.

The computing machine 100 can be embodied in any one of the followingcomputing devices: a computing workstation; a desktop computer; a laptopor notebook computer; a server; a handheld computer; a mobile telephone;a portable telecommunication device; a media playing device; a gamingsystem; a mobile computing device; a device of the IPOD family ofdevices manufactured by Apple Computer; any one of the PLAYSTATIONfamily of devices manufactured by the Sony Corporation; any one of theNintendo family of devices manufactured by Nintendo Co; any one of theXBOX family of devices manufactured by the Microsoft Corporation; or anyother type and/or form of computing, telecommunications or media devicethat is capable of communication and that has sufficient processor powerand memory capacity to perform the methods and systems described herein.In other embodiments the computing machine 100 can be a mobile devicesuch as any one of the following mobile devices: a JAVA-enabled cellulartelephone or personal digital assistant (PDA), such as the i55sr, i58sr,i85s, i88s, i90c, i95cl, or the im1100, all of which are manufactured byMotorola Corp; the 6035 or the 7135, manufactured by Kyocera; the i300or i330, manufactured by Samsung Electronics Co., Ltd; the TREO 180,270, 600, 650, 680, 700p, 700w, or 750 smart phone manufactured by Palm,Inc; any computing device that has different processors, operatingsystems, and input devices consistent with the device; or any othermobile computing device capable of performing the methods and systemsdescribed herein. Still other embodiments of the computing environment101 include a mobile computing device 100 that can be any one of thefollowing: any one series of Blackberry, or other handheld devicemanufactured by Research In Motion Limited; the iPhone manufactured byApple Computer; any handheld or smart phone; a Pocket PC; a Pocket PCPhone; or any other handheld mobile device supporting Microsoft WindowsMobile Software.

With reference to FIG. 2, one embodiment of a system for displayinggraphics and other application output at a remote computing machine 202is shown and described. As shown, the system includes a remote computingmachine 202 communicating with one or more local computing machines 204.The remote computing machine 202 and the local computing machines 204may be embodiments of the computing device 102. In some embodiments, wemay refer to the remote computing machine 202 as a client device and thelocal computing machines 204 as servers. The system also includes one ormore GPUs 216, 216′or GPU appliances, in connection with one or more ofthe computing devices. The local computing device 204 communicates tothe remote computing machine 202 via an appliance 206. The appliance 206may be a proxy, gateway, intermediary, or any network node. In someembodiments, the appliance 206 may be a GPU appliance or can include aGPU appliance.

The local computing machine 204 provides application output to theremote computing machine 202 in response to the executing of one or moreapplications 208 at the local computing machine 204. The local computingmachine 204 can also provide desktop data for rendering at the remotecomputing machine 202. The application/desktop data can include imagedata and/or windowing data for processing and/or rendering at the remotecomputing machine 202. In such embodiments, an Application/DesktopDelivery System 210 communicates the application/desktop data to theremote computing machine 202. The application/desktop data, responsiveto the rendering process at the remote computing machine 202, isdisplayed via a Remote Application Presentation Window 212 at the remotecomputing machine 202. In some embodiments, the local computing machine204 is not executing the applications 208 and instead an application 208is streamed to the remote computing machine 202 for local execution.

Each of the remote computing machine 202 and local computing device 204may include a main processor 121, 121′, a Graphical Processing Unit(GPU) 216, 216′, memory element 122, 122′, and a three-dimensionalGraphics Library 220, 220′. The main processor 121, 121′ may be anyembodiment of the CPU or Main Processor 101 described above inconnection with FIGS. 1B and 1C. The GPU can in some embodiments be ahardware component dedicated to processing graphics commands, while inother embodiments, the GPU can be a set of executable commands, orexecutable program able to process graphics commands. The memory element122 may be any embodiment of the main memory 122 described above inconnection with FIGS. 1B and 1C. The three-dimensional Graphics Library220, 220′ may be a library associated with Direct3D, OPEN GL or otherthree-dimensional graphics Application Program Interface (API). Althougheach of these elements may provide the same or substantially the samefunctionality between the remote computing machine 202 and localcomputing device 204, this can provide redundancy as well as theflexibility to allow the rendering location to be located between theremote computing machine 202 and local computing device 204.

The remote computing machine 202 includes a client agent 214 thatprovides certain functionality for use in determining where to rendergraphics data. The local computing machine 204 can also include a serveragent (not shown) for providing substantially similar functionality forthe local computing machines 204, appliance 206, and network. In otherembodiments, a single “graphics determination agent” resident somewherewithin the system provides the below-described functionality. Factorsconsidered in making a determination as to where to render graphicsinformation, include, but are not limited to: the type of graphicsprimitives being used (e.g., OPEN GL, DirectX, Direct3D, 3DNOW, GDI,MILCORE, and so on); the GPU capabilities of the remote computingmachine 202, the local computing machines 204, the appliance 206, andthe GPU appliance; the type, status and delivery speed of the network;the type of application being executed (e.g., CATIA, AUTOCAD, WINDOWSMEDIA PLAYER); and the type of remote access session being used (e.g.,Terminal Services, Citrix Presentation Server, Citrix Desktop Server,application streaming and so on).

In various embodiments, the selection of the rendering location canchange from time-to-time, that is, the selection of the renderinglocation is dynamic in nature. This selection, and the determination ofthe selection, can be made on a real-time basis, a per-session basis, aper-user basis, or according to any other mechanism. For example, aninitial scan of the system configuration may determine to use the remotecomputing machine's 202 GPU 216′ to render the graphics. At a latertime, conditions may change and thus it may provide a better end-userexperience to render the graphics at the local computing machines 204and/or the GPU appliance 250. However, portions of the session can berendered using different methods. For example, portions of a desktop orapplication image data can be rendered using a rendering process at thelocal computing machine 204 while other portions can be rendered using“client-side” rendering. Further, the selection of the renderinglocation can occur according to particular graphics operations (e.g.,vector drawing, texture mapping, and the like). The rendering locationcan differ based on the type of graphics operation. Different renderingoptions can be used at the same time for different operations. Forexample, a spinning globe may be rendered on the remote computingmachine 202, but the background static graphics may be rendered on thelocal computing machine 204. Also, selection can occur based on apattern of graphics operations (e.g., if many small shaded objects aredrawn sequentially).

In some embodiments, the selection of which local computing machine torun the desktop or application, can be based on what types of graphicsrendering are most efficient. The “load balancing” selection can bebased on the graphics rendering policy evaluation. For example, if aremote computing machine 202 does not have a GPU 216′ with thecapability or capacity to handle the image data, but the applicationneeds it, then a local computing machine 204 is selected that has anavailable and suitable GPU 216.

In one embodiment, network bandwidth latency, remote computing machinecapabilities, the application type of the application 208, past history,and any other graphics performance policy rules may be evaluated. Ifperforming rendering at the local computing machine 204 is determined tobe the most efficient or suitable, a local computing machine 204 that iscapable of performing such rendering would be selected. Other loadbalancing policies can affect local computing machine selection as well(e.g., user, location, date, time, application licensing, CPU, diskusage, memory, etc). More intense graphical usage may be routed tocomputing machines that have a better performing GPU or have more GPUsin comparison with other local computing machines. In the case ofsharing the server and GPUs, the load on the available GPUs can be takeninto account.

In Windows Vista, a DirectX application sends graphics to a Direct3Dgraphics runtime module which cooperates with a user mode driver and agraphics kernel to render the DirectX graphics commands on availablehardware. When a DirectX application executes in Terminal Services,there may be a degradation in the execution of the DirectX applicationand the generation of enhanced graphics. One cause of the possibledegradation in performance is that the application, when executing inTerminal Services, no longer has access to graphics rendering hardwaresuch as a graphics processing unit (GPU). Thus, the DirectX applicationrelies on software, executing in the Direct3D runtime and kernel, torender the DirectX graphics commands into graphical data. Loss of accessto graphics rendering hardware may therefore degrade the performance ofthe DirectX application. A similar situation may also occur in the sameor another operating system using OPEN GL or other three-dimensional APIin place of Direct3D.

Illustrated in FIG. 3 is an embodiment of a system that can alleviatethe potential issues associated with executing a DirectX application interminal services, as described above. The system comprises client-sideor terminal-side 421 components and server-side 420 components. Includedin this system is a three-dimensional Rendering Application 414 incommunication with a three-dimensional runtime module 404′. A three-dimensional proxy 406 is included and is in communication with athree-dimensional Graphics Runtime module 404.

Further referring to FIG. 3 and in more detail, described is a systemthat, in some embodiments, permits graphics applications, such asDirectX applications, executing in terminal services to use the graphicshardware on the server as opposed to software executing in thethree-dimensional graphics runtime module 404. Such a system can focuson the use of DirectX and Windows Presentation Foundation applications.In some of these embodiments, the three-dimensional renderingapplication 414 may be a Direct3D rendering application, thethree-dimensional runtime modules 404, 404′ may be Direct3D runtimemodules, and the three-dimensional proxy 406 may be a Direct3D proxy,for example. Any of the components of the system may be componentsassociated with Direct3D, OPEN GL or other three-dimensional graphicsAPI.

The three-dimensional Rendering Application 414 transmits commands tothe Three -dimensional Runtime Module 404′. The three-dimensionalRendering Application 414 acts as a proxy for the application in thesession by receiving commands forwarded from the three-dimensional proxy406. In operation, as an application attempts to drive thethree-dimensional system to render a scene, the three-dimensional proxy406 reroutes its graphics commands to the three-dimensional GraphicsRendering Application 414, which, in turn, sends these commands to itsthree-dimensional Graphics Runtime module 404′ as if it were therequesting application. Further, the three-dimensional proxy 406 allowsthe application to operate just like it normally would whencommunicating with the three-dimensional Graphics Runtime module 404 inthe session or server space.

In one embodiment, the three-dimensional proxy 406 interceptsthree-dimensional graphics calls (e.g., Direct3D calls) and routes themto the three-dimensional Rendering Application 414 running in theconsole or client session. Also, the three-dimensional proxy 406initiates communication between itself and the three-dimensionalGraphics Manager 412. In some embodiments, the three-dimensionalGraphical proxy 406 initiates communication between itself and thethree-dimensional Graphics Manager 412 via an Image Transfer Service410. The three-dimensional Graphics Manager 412, in response to theinitiation, creates the three-dimensional graphics rendering applicationservice. In some embodiments, the three-dimensional Graphical proxy 406hooks or intercepts the real runtime calls but may only target graphicscalls that would have been handled by the display driver 408.

In one embodiment, a three-dimensional Graphics Manager 412 creates thethree-dimensional Rendering Application 414 service and establishes acommunication path between the three-dimensional Rendering Application414 and the three-dimensional Graphics proxy 406 which requested serverhardware support.

In summary, the embodiment of the architecture described above withreference to FIG. 3 provides access to the graphics hardware of theserver 420 (e.g., a GPU or GPU appliance) in a terminal service session.As a result, the process of remote-transporting bitmapped graphics tothe remote computing machine or client 421 may be an improvement overrendering the bitmaps in software.

In another embodiment, the Application 502 may provide a “desktop” suchas the AERO desktop provided by WINDOWS Vista and the remote computingmachine 421 is accessing a remote desktop on the local computing machineor server 420. In some embodiments, the client's 421 GPU is used torender either the OPEN GL or Direct3D scenes associated with thedesktop. In one aspect, the transmission of the graphics informationfrom the local computing machine 420 to the remote computing machine 421can be improved relative to other scenarios.

With reference to FIG. 4, a diagram shows one embodiment of anarchitecture that outlines the components and data flow between thecomponents. In this architecture, the layering between the displaydriver and application is thin so as to limit the overhead anddevelopment effort.

In one embodiment, the architecture includes an Application 402. TheApplication 402 may be a Desktop Windows Manager (DWM) module. In oneembodiment, the Application 402 is a Windows Presentation Foundation(WPF) application that acts as a window manager. The Application 402 caninteract with a D3D9Ex interface. The Application 402 is responsible forrendering the non-client portion of the windows and the transitioneffects that affect the windows. In one embodiment, the Application 402is provided by the system or application that uses Direct3D, OPEN GL orother three-dimensional graphics API. In some embodiments, theApplication 402 uses Direct3D to create a desktop. The Application 402provides the 3D geometry to render the desktop scene on a display. Insome aspects, the Application 402 drives the three-dimensional system(e.g., Direct3D system) to render the scene associated with the desktop.In such embodiments, the Application 402 may have no specific function,but rather is the source of the three-dimensionalgeometry.

The architecture also includes a three-dimensional Graphics Hook module504. In one embodiment, the three-dimensional Graphics Hook module 504provides functionality to detect screen updates by intercepting aPresent call that causes the back buffer to be switched with the frontbuffer. Once the hook detects the Present call, the hook passes thatcall through to one or more three-dimensional Graphics Libraries 220,220′. The hook collects the window information for the application andthe image data from the graphics card via the Display Driver 506. Thehook then presents both the image data and window information to theImage Transfer Service 508.

In some embodiments, the three-dimensional Graphics Hook module 504provides functionality related to: detecting screen updates; hooking thethree-dimensional data to detect screen updates; presenting the ImageTransfer Service 508 with data such as image data from the graphics cardand information for the context. Also, the three-dimensional GraphicsHook module 504 provides a reliable hooking environment with limitedoverhead. The image collection process, via the three-dimensionalGraphics Hook module 504, collects portions of the screen that are beingupdated through the Present call.

As discussed above, the architecture includes three-dimensional GraphicsLibraries 220, 220′. In one embodiment, a three-dimensional GraphicsLibrary 220 is a dynamic link library (dll). The three-dimensionalGraphics Libraries 220, 220′ provides functionality related to bufferingof commands and geometry data from the Application 502, to be sent to ahardware display driver, such as Display Driver 506. In someembodiments, the three-dimensional Graphics Libraries 220, 220′ packagesthe three-dimensional graphics calls into buffers for bulk transfer. TheDisplay Driver 506 can be an existing hardware driver on the server 420.The Display Driver 506 operates as a conventional hardware displaydriver, and also allows the three-dimensional Graphics Hook module 504to collect image data off the graphics card. The Display Driver 506 alsorenders the associated scene into a bitmap and delivers the image to thethree-dimensional Graphics Hook module 504. In one embodiment, thethree-dimensional Graphics Hook module 504 sends a request to theDisplay Driver 506. Responsive to the request, the Display Driver 506transmits image data to the three-dimensional Graphics Hook module 504.

The architecture also includes an Image Transfer Service module 508, asmentioned above. This module communicates with the three-dimensionalGraphics Hook module 504. The Image Transfer Service module 508 may alsocommunicate with a Winstation Driver (not shown). The Image TransferService module 508 includes functionality for an image transport virtualchannel that delivers image data to the client 421. In variousembodiments, the Image Transfer Service module 508 implements one ormore compression schemes, for example, aggressive compression. As usedherein, aggressive compression refers to a compression process thatlowers the quality of images during animation routines to allow morethroughput at the cost of image quality. This process may improve framerefresh rates by reducing the level of detail in the image duringanimation. The final frame of the animation, or each animation section,may be determined and sent at a higher quality.

In some embodiments, the Image Transfer Service module 508 requests theimage data from the three-dimensional Graphics Hook module 504, and mayimplement any combination of aggressive compression, caching, andimplementing a new image virtual channel. Further, the Image TransferService module 508 can receive Window notifications. Additionally, theImage Transport Service module 508 aides in managing the network trafficto achieve maximum frame rates or rates meeting a certain service level.

The architecture may also include a Winstation Driver module (notshown), as mentioned above. In some embodiments, the Winstation Drivermodule aides in implementing the virtual channel multiplexing to asingle connection. This includes the implementation of optimizations fornetwork traffic and virtual channel priorities queues.

Illustrated in FIG. 5A is an embodiment of a method 600 for remotelyproviding three-dimensional graphics commands to a remotely locatedmachine. The three-dimensional graphics are first intercepted (step 602)by an agent, and at least a portion of the three-dimensional graphicscommands are encoded by the agent (step 604). A frame is then createdthat comprises all encoded portions of the three-dimensional graphicscommands (step 606). The frame is compressed (step 608) and thecompressed frame is transmitted to a remote computing machine (step610).

Further referring to FIG. 5A and in more detail, the method 600 in someembodiments is carried out by an agent executing on any one of: a firstcomputing machine; a computer; a computing device; a server; a serverfarm; a distributed computing system; a network; or any other computingdevice embodiment, or network embodiment described herein. In many ofthese embodiments, the agent executes on the same computing machine,network or appliance as the one on which thethree-dimensionalapplication is executing. While in some embodiments theagent is a virtual object executing on a local machine, the agent mayalso be a function, set of commands, or other computing object executingon a computing machine in a distributed computing environment.

Three-dimensional graphics commands are generated by an applicationexecuting on a computing machine, these three-dimensional graphicscommands are in some embodiments intercepted (step 602) by an agent orother computing object executing on the same machine as the machine onwhich the application executes. In one embodiment, the agent interceptsthe three-dimensional graphics by hooking into the present call of theapplication. By doing this, the agent can determine, via applicationprogram interfaces, when the application sends graphics commands to athree-dimensional graphics runtime module. Once the agent knows that theapplication has made a call to the three-dimensional graphics runtimeapplication program interface, the agent can then intercept the call andredirect it to the client so that the three-dimensional graphics can bereplayed on the client. In one embodiment, the agent sends all commandsdirectly to the client, while in other embodiments a portion of thethree-dimensional graphics commands are rendered on the server and aportion of the three-dimensional graphics commands are rendered on theclient. When commands are sent to the client, in one embodiment they canbe any combination of compressed, optimized, batched, encoded andtransmitted across a network from a local computing machine or server toa remote computing machine or client computing machine. In suchembodiments, what is transmitted to the client computing machine orremote computing machine is a frame containing any combination of:three-dimensional drawing commands; a render state; and resources neededto render or draw a two-dimensional graphical representation or atwo-dimensional surface render target that represents the projection ofthree-dimensional onto a portion of a two-dimensional plane or drawingsurface. Thus, in these embodiments, the server or local computingmachine transmits frames to the client computing machine or remotecomputing device, which then maps the commands to a three-dimensionalgraphics runtime module that accepts the frames and rendersthree-dimensional data from the accepted frames.

In one embodiment, a portion of the three-dimensional graphics commandsare encoded (step 604) by the agent or by another system object,function or agent that is in communication with the agent. In oneembodiment, there can be multiple three-dimensional structuresrepresented by the three-dimensional graphics commands. For example, thethree-dimensional graphics commands can include any of the following:vertex buffers; index buffers; textures; and surfaces. Vertex buffersare buffers that contain information about the vertices associated witha particular aspect of the resultant three-dimensional image. In someembodiments, the vertex buffer can include a structure for each vertexthat includes any of the following information: three-dimensionalcoordinates; depth coordinates; texture mapping coordinates; color; andany other characteristic of a vertex able to be stored in the vertexbuffer. Index buffers are another type of buffer that includes indicesinto vertex buffers. Textures and surfaces are representative ofphotographic or synthetic two-dimensional raster data.

In some embodiments, the portion of the three-dimensional graphicscommands that is encoded is a portion that corresponds to a particularstructure. For example, one portion could include all vertices, anotherportion could include all indices, and still another portion couldinclude all textures and surfaces. This example is illustrative of thedifferent portion segments that could be created, in other embodimentsthese segments may be numbered or may include multiple structures suchas a portion containing both vertices and indices. Once a portion of thethree-dimensional graphics commands is identified, that portion isencoded with any of the following types of codecs: Huffman encoding;adler-16, adler-32; CRC-16; CRC-32; Fletcher-16; Fletcher-32;photographic lossy codecs; synthetic lossy codecs; JPEG; HD Photo;2DRLE; PNG; HD Photo; and any other encoding suitable for the structurebeing encoded. In one embodiment, vertices and indices are encoded usingHuffman encoding, while textures and surfaces would be encoded useeither JPEG, HD Photo, 2DRLE, PNG, or HD Photo.

In one embodiment, once the portion of the three-dimensional graphicscommands has been encoded, a frame is created (step 606) that compriseseach encoded portion of the three-dimensional graphics commands. Theseencoded portions can in some embodiments be any of the structuresdescribed herein encoded using any of the codecs described herein. Forexample, the three-dimensional graphics commands could include each of avertex buffer, an index buffer and textures. In one embodiment, theframe could include vertices encoded using Huffman encoding, indicesencoded using Huffman encoding, and textures encoded using a syntheticlossy codec. The frame, in some embodiments, comprises the elements ofthree-dimensional drawing commands; a render state; and resources neededto render or draw a two-dimensional graphical representation or atwo-dimensional surface render target that represents the projection ofthree-dimensional onto a portion of a two-dimensional plane or drawingsurface.

Once the frame is created, the agent may in some embodiments compressthe frame (step 608) prior to transmission. In some embodiments, acompressor may compress the frame, while in other embodiments, the framemay be compressed after being put through a compression algorithm. Stillother embodiments include transmitting a frame that has not beencompressed, or transmitting a frame having a portion of compressed dataand a portion of non-compressed data.

The local computing machine or server transmits the compressed frame tothe remote computing machine or client (step 610) so that a renderingagent, application, or unit may render three-dimensional data from thegraphical data contained within the frame. In some embodiments, theframe may be de-compressed prior to rendering.

Illustrated in FIG. 5B is an embodiment of a method 650 for encodingportions of graphical commands as indicated in step 604 of the method600 described in FIG. 5A. The method 650 commences once the method 600enters step 604, by determining where the three-dimensional graphicscommands include vertex commands (step 652). If the three-dimensionaldata does contain three-dimensional graphics commands that includevertex commands, then the portion of the three-dimensional datacorresponding to the vertex commands is encoded with a codec (step 654).A determination is then made as to whether the three-dimensionalgraphics commands include index commands (step 656). If thethree-dimensional data does contain three-dimensional graphics commandsthat include index commands, then the portion of the three-dimensionaldata corresponding to the index commands is encoded with a codec (step658). A determination is then made as to whether the three-dimensionalgraphics commands include textures or surfaces (step 660). If thethree-dimensional data does contain three-dimensional graphics commandsthat include textures or surfaces, then the portion of thethree-dimensional data corresponding to the textures or surfaces isencoded with a codec (step 662).

Further referring to FIG. 5B, and in more detail, method 650 is asub-process, sub-function or sub-method of method 600 described in FIG.5A. The method 650, in one embodiment, parses through the threedimensional graphics commands to determine whether different types ofcommands are included in the three dimensional graphics command set.Method 600 is redirected to the method 650 described in FIG. 5B, andwhen the method 650 in FIG. 5B has finished executing, method 600 picksup where it left off. Alternatively, the method 650 described in FIG. 5Bcould be a part of step 604 described in FIG. 5A.

Illustrated in FIG. 6A is an embodiment of a method 700 for determiningwhere to render three-dimensional graphical data. An agent or systemobject intercepts three-dimensional graphics commands (step 702)generated by an application using three-dimensional graphics. Beforerendering or transmitting the intercepted commands, the characteristicsof the remoting system are analyzed (step 704) to determine a renderinglocation (706). Upon determining a rendering location (706), the agentor system object induces the application to reinitialize a context fordetermining where to render three-dimensional data (step 708). Once theapplication has identified the determined rendering location as thelocation for rendering, the agent or system object transmits thethree-dimensional graphics commands the rendering location for rendering(step 710).

Further referring to FIG. 6A, and in more detail, the method 700 fordetermining where to render the three-dimensional data can be carriedout by an agent executing on a server or local computing machine.Alternatively, the method can be carried out by a set of commands, orprogram executing on a server or local computing machine and taking asinput the three-dimensional graphic command output from the application.Still other embodiments can include a rendering manager, a renderingagent, or some other virtual object able to analyze system resources anddetermine where to render three-dimensional data.

In one embodiment the three-dimensional graphics commands areintercepted by the remoting agent (702), or any other embodiment of anentity that intercepts three-dimensional graphics commands. Interceptingthe graphics commands can be accomplished, in some embodiments, byhooking into a call made by the desktop windows manager.

After intercepting the three-dimensional graphics commands, thecharacteristics of the remoting system are analyzed (704). The remotingsystem can include any of the following components: one or more clientcomputing devices such as those described herein; one or more serverssuch as those described herein; a server farm; one or more appliancessuch as those described herein; and networks over which each of thesecomponent communicate and transmit data. In one embodiment, thenetwork(s) used within the remoting system can be any of the networksdescribed herein. Further embodiments include at least one serverexecuting an application, and at least one client computing machineaccessing the server to receive graphical data representative of theapplication's output. In such an embodiment, communication between theclient and the server could be facilitated by a remote access protocol.Characteristics of this system can include in some embodiments: the typeand/or speed of the graphics processing unit included on the server; thetype and/or speed of the graphics processing unit included on the clientcomputing device; the type and/or speed of the central processing unitincluded on the server; the type and/or speed of the central processingunit included on the client computing device; the application executingon the server; the load on the server; the load on the client computingdevice; the load on the network; the amount of available bandwidth overwhich to transmit the three-dimensional graphics commands; apredetermined setting indicative in part of where to render thethree-dimensional data; the number of applications on the serverrendering three-dimensional data; the lack of a graphics processing uniton either the server or the client computing device; and any othercharacteristic which may be used to determine a rendering location. Inone embodiment, various characteristics are aggregated to produce asingle number for each of the rendering locations, such that therendering location can be chosen by comparing the numbers to see eitherwhich number is the highest compared to the others, or which number isthe lowest compared to the other numbers. Still other embodimentsinclude weighting various characteristics according to their tax on thesystem and choosing where to render based on the remoting location thatcosts the least amount of resources.

The results generated during the analysis of the characteristics of theremoting system are used, in some embodiments, to determine a renderinglocation (step 706). In one embodiment, the remoting location can be oneither the server or the client computing machine. If the renderinglocation is the server, then the server renders the three-dimensionaldata from the three-dimensional graphics commands and transmits therendered graphical data to the client computing machine. If therendering location is on the client, then the server transmits thethree-dimensional graphics commands to the client such that the clientreceives the commands and renders three-dimensional data from them.Other embodiments include a remoting location that can either be agraphical processing unit on the server or client that is a hardwarecomponent, or a graphical processing unit on the server or client thatis a software component. Should the result of the analysis of thecharacteristics of the remoting system demonstrate that rendering thedata on a software graphical processing unit would be best, then theremoting location can be a location having a software-based graphicalprocessing unit.

In one embodiment, the agent induces the application to reinitialize itscontext for determining where to render the three-dimensional data (step708) by reporting to the application a device state removed. In otherembodiments, the agent may report to the application a device statelost. When the application is notified that the device was removed, theapplication can in some embodiments re-determine where to render data.In one embodiment, the agent can take advantage of this re-determinationor the reinitialization of a context for determining where to renderthree-dimensional data, by telling the application that it should renderthe three-dimensional data in a particular location. Some embodimentsconsist of telling the application that it should render thethree-dimensional data at the determined rendering location.

Once the application is re-configured to render to the renderinglocation, the three-dimensional graphics are sent to the renderinglocation to be rendered.

Illustrated in FIG. 6B is a method 730 for rendering three-dimensionalgraphics commands at a particular location. In one embodiment, themethod 730 includes determining to render a portion of three-dimensionalgraphics commands in a first rendering location (step 732), analyzingthe characteristics of a remoting system (step 734) and determining afirst rendering location and reinitializing the application context toselect the determined first rendering location (step 736). The first setof three-dimensional data is then rendered, at the rendering locationfrom the first portion of the three-dimensional graphics commands (step738). The characteristics of the remoting system are then re-evaluated(step 740), a second rendering location, is determined and theapplication context is reinitialized to identify a second renderinglocation as the location where the data will be rendered (742). Once thesecond rendering location is determined, the three-dimensional graphicscommands and the rendered first set of three-dimensional data aretransmitted to the second rendering location (step 744).

Further referring to FIG. 6B and in more detail, in one embodiment, themethod 730 comprises determining to render a portion of thethree-dimensional graphics commands (step 732). This portion can in someembodiments, be determined based on any of the following factors: typeof graphic element (i.e. vertices, indices, textures); size of thegraphical data; a transaction; or any other means by which to segmentthe three-dimensional graphics commands. In one embodiment, eachtransaction creates a new portion of three-dimensional graphics commandssuch that a determination as to where to render the graphics commandsmust be made each time a transaction takes place.

In one embodiment, a portion of the three-dimensional graphics commandsis rendered and the resulting rendered graphical data remains on theserver. In such an embodiment, a second portion of three-dimensionalgraphics commands is identified and a new determination is made as towhere to render the second portion of three-dimensional graphicscommands. In one embodiment, the method includes determining to render aportion of the three dimensional graphics in a first rendering location(step 732). Some embodiments may include adhering to a policy enginewhich may dictate that certain types of graphics, graphics primitives,or three dimensional graphics types should always be rendered in aparticular location whether that be on a local computing machine, aremote computing machine, the network or a third computing machine. Inother embodiments, a rendering agent may determine that certain types ofgraphics, graphics primitives, or three dimensional graphics typesshould be rendered in a particular location either because of theirsize, or for other reasons. For example, the rendering agent may specifythat certain three dimensional graphics be rendered on the localcomputing machine because those types of graphics can only be renderedby the graphics processing unit included on the local computing machine.In such an example, the graphics would be rendered on the localcomputing machine and the rendered graphics or bitmaps would be sent toa client machine or to any other remote computing machine. Should thethree dimensional graphics be included within a graphics schemeincluding graphics commands that a remote computing machine couldrender, the rendering agent may determine to transmit a graphics packageto the remote computing machine, that includes the three dimensionalgraphics already rendered by the server and the graphics commands. Theremote computing machine could then render the graphics commands at theremote computing machine.

Illustrated in FIG. 7 is a flow diagram that illustrates an embodimentfor method 760 of computing and caching a hash from a three-dimensionaldata set. Computing and caching the hash may include a determination ofwhether a three-dimensional resource has been locked by an application(step 762). An identification is made of a first three-dimensional dataset loaded into the three-dimensional resource (step 764). The method760 may further include detecting if the application performed anoperation on the three-dimensional resource (step 766). A hash iscomputed on the loaded three-dimensional data (step 768), and the hashis stored in a repository (step 770). A determination is then made as towhether there are additional data sets (step 772), thereby causing themethod 760 to either repeat or terminate.

Further referring to FIG. 7 and in more detail, the method 760 mayinclude determining whether an application has locked athree-dimensional resource (step 762). In some embodiments, thethree-dimensional resource may include vertices, indices, shaders,textures, surfaces, or any other similar three-dimensional data type. Anapplication may lock a three-dimensional resource while modifying datawithin the source or loading data into the resource in order to preventprocesses from reading the changing resource. In some embodiments, thedetermination that an application has locked a three-dimensionalresource may be made by examining flags, counters, synchronizationmonitors, or queues for the data, or any other method that allows aprocess to determine that a resource has been locked for writing. Infurther embodiments, the three-dimensional resource locked by theapplication may be one used by the application, while in otherembodiments, the application may lock a three-dimensional resource usedby another application. The three dimensional resource can, in someembodiments, be any one of the following types of resources: the driverfor a graphics processing unit; or any other resource used inconjunction with a graphics processing unit or graphics processingalgorithm, set of instructions or software.

The method 760 may further include a step of identifying athree-dimensional data set loaded into the three-dimensional resource(step 764). The three-dimensional data set may be a firstthree-dimensional data set or may be a second three-dimensional dataset, a third three-dimensional data set, or any further data setdepending on the number of data sets loaded into the three-dimensionalresource. In some embodiments, the three-dimensional data sets may havevariable sizes, while in other embodiments, any or all of them may havesubstantially similar sizes. The sizes of each data set may, in someembodiments, be determined at least in part by a transaction carried outby the application. In some embodiments, the transaction may comprisecopying data into the three-dimensional resource. In furtherembodiments, the transaction may have an end event unlocking thethree-dimensional resource, the end event thus can indirectly determinethe size of the data set. In other embodiments, the transaction may haveboth a start event and an end event, and the data set size may bedetermined by the length of time between the start event and end event.In such embodiments, the start event may comprise locking thethree-dimensional resource, and/or the end event may comprise unlockingthe three-dimensional resource.

In one embodiment, the method 760 includes detecting that theapplication performed an operation on the three-dimensional resource(step 764). This may include detecting that an application has locked orunlocked a three-dimensional resource by examining or identifying flags,counters, synchronization monitors, or queues, or any other method thatallows a process to determine that a resource has been locked orunlocked for writing. In some embodiments, detection that an operationhas been performed on the three-dimensional resource may be made merelyby observing that the three-dimensional resource has been unlocked aftera period of being locked. In other embodiments, the application may bemonitored to determine what operations have been performed.

In one embodiment, the method 760 includes a step of computing a hash onthe loaded three-dimensional data (step 768). The hash functionperformed may be any hash function or checksum algorithm that hassufficiently few collisions for the type of data used, such as adler-16,adler-32, CRC-16, CRC-32, Fletcher-16, Fletcher-32, or any other CRC ornon-CRC hash function., or a hash function that produces minimal orsubstantially no collisions for the type of data on which the hashfunction is applied; or any other hash function able to produce resultssuitable for the systems and methods described herein.

In one embodiment, the method 760 includes a step of storing the hash ina repository (step 770). In some embodiments, the repository may be abuffer, a cache, a data store, a flat file, or any other format, and maybe implemented in RAM, flash memory, a hard disk, or any other storagemedium. In some embodiments, the repository may stored on a localcomputing device, while in other embodiments, the repository may resideon a second computing device, accessible over a network. In furtherembodiments, the repository may be purged by using an aging algorithm.The aging algorithm may be any page replacement algorithm, such as LeastRecently Used (LRU), Not Recently Used (NRU), Active Replacement Cache(ARC), First-in, First-out (FIFO), Least Frequency Usage (LFU), or anysimilar algorithm that allows the repository to evict entries on anefficient basis.

In one embodiment, the method 760 includes a determination of whetheradditional data sets exist in the three-dimensional resource (step 772).If such a determination is made, then method 760 will further includeidentifying further three-dimensional data sets loaded into thethree-dimensional resource, detecting further operations performed bythe application to the three-dimensional resource, computing hashes ofthe further three-dimensional data sets, and storing the hashes of thefurther three-dimensional data sets within the repository.

In one embodiment, the method 760 may further include eliminating aportion of the first three dimensional data set in response to detectingthat the application performed an operation. The portion of the threedimensional data set that may be a result of a repetitious transaction.In other embodiments, the computing machine only maintains and storescomputed hashes and drives a remote computing machine cache bydetermining which data to throw out to make room for new data on a missand maintain an overall max size. Determining which data to throw outcan be based in part on an aging algorithm such as a Least FrequencyUsage algorithm. Such an algorithm increments a counter on a hit andwhen space is needed on a miss, the entries that have the lowest counterand size that produces the least impact to bandwidth are evicted.Further embodiments can include a caching system where there ispersistent caching of high frequency resources on a client, computingmachine or remote computing machine's persistent storage disk. Thus, insuch an embodiment, textures and surfaces that are loaded during aninitial setup during an application's start-up or new scene are loadedmore quickly so that the application restarts more quickly than if therewas no persistent caching of high frequency resources. Further, the useof high frequency dynamic data can be more efficient because hashes ofthe dynamic data can be cached and saved by a client, computing machineor remote computing machine when an application terminates.

In other embodiments, the application can transfer, move, copy, load, orotherwise cause data to be put into the three dimensional resource.Still other embodiments include an application that transfers data to anintermediary application configured to load, copy or transfer the datainto the three dimensional resource. In such an embodiment, theintermediary application can exist on the same computing device as theapplication, a computing device remote from the computing device onwhich the application executes, or any other virtual or physicalcomputing device or appliance in communication with the computing deviceon which the application executes.

In one embodiment, the method 760 is repeated only when an applicationor agent detects that the three dimensional resource is being used byany application, or in other embodiments, by a particular application.Other embodiments include repeating the method 350 only when anapplication or agent detects that the three dimensional resource has haddata copied or loaded into it, and the three dimensional resource has oris being used by an application. In yet another embodiment, the method760 is repeated only when it is determined that data different frompreviously cached data is loaded or copied into the three dimensionalresource. Other embodiments include repeating the method 760 only when aparticular sequence of actions is detected, e.g. lock the threedimensional resource, load or copy data into the three dimensionalresource; and use the three dimensional resource. Still more embodimentscan include a method 760 that is periodically repeated.

Illustrated in FIG. 8A and 8B are embodiments of screenshots of a methodfor detecting a drawing region within a frame. A frame 790 may contain afirst three-dimensional FIG. 792 in a first presented bitmap in FIG. 8A.In a second presented bitmap in FIG. 8B, the frame 790 contains thefirst three-dimensional FIG. 792, and a second three-dimensional FIG.794. The second three-dimensional FIG. 794 is within a region 796 thatis a subset of the frame 790.

Referring to FIG. 8A and in more detail, the frame 790 may also bereferred to as a scene, a default viewport, a canvas, an image, abitmap, a screen, a surface, or any other descriptive term for adisplayed three-dimensional graphic. Within the frame 790 may be a firstthree-dimensional FIG. 792. The first three-dimensional FIG. 792 mayinclude vertices, indices, shaders, textures, surfaces, or any othersimilar three-dimensional data type. In some embodiments, the firstthree-dimensional FIG. 792 may not exist and the frame 790 may be empty.In other embodiments, the first three-dimensional FIG. 792 may beaccompanied by other three-dimensional figures, not illustrated.

Referring to FIG. 8B and in more detail, in one embodiment, a secondthree-dimensional FIG. 794 is added to the frame 790. In someembodiments, multiple three-dimensional figures, not illustrated, may beadded simultaneously. In other embodiments, the second three-dimensionalFIG. 794 may be have been present in the previously presented frame 790,and may be removed. In still other embodiments, multiple figures,including the second three-dimensional FIG. 794, the firstthree-dimensional FIG. 792, and other three-dimensional figures notillustrated may be removed, added, or modified.

In one embodiment, the second three-dimensional FIG. 794 is containedwithin a sub-region 796 of the frame 790. In the illustrated example,the portion of frame 790 outside of the region 796 does not change fromthe previously presented bitmap in FIG. 8A to the newly presented bitmapin FIG. 8B. The region 796 may be referred to as a drawing region, aviewport, a canvas or canvas portion, a screen section, a section of theframe, a view-clipping region, a view-clipping rectangle, or any otherterm that is descriptive of a sub-portion of the frame 790. In someembodiments, multiple three-dimensional figures may be added to thenewly presented bitmap, and they may be grouped within multiplenon-contiguous regions (not illustrated). In other embodiments, multiplefigures may be grouped within a single region. The region 796 may also,in some embodiments, represent a sub-portion of the frame 790 where athree-dimensional figure or figures have been removed or modified. Insome embodiments, the region 796 may be designated by a function call,such as the SetViewPort method of the DIRECT3D API developed by theMicrosoft Corporation, or the g1Viewport method of the Open GraphicsLibrary API (OPENGL), developed by Silicon Graphics, Inc., or anysimilar method or function call that may define a region or sub-regionof the frame 790 of a render target.

Illustrated in FIG. 9 is a method 800 for detecting a dirty regionwithin a frame. Detecting a dirty region may include intercepting afunction call generated by a three-dimensional application (step 802).The intercepted function call may be used at least in part to identify adrawing region within a frame (step 804). A determination may be madethat at least a portion of the graphics commands associated with theidentified drawing region have changed (step 806). The changed portionof the graphics commands within the identified drawing region are thenextracted (step 808).

Further referring to FIG. 9 and in more detail, the method 800 mayinclude a step of intercepting a function call generated by athree-dimensional application (step 802). Intercepting a function callmay also be referred to as hooking, monitoring, listening, capturing, orany other term descriptive of a third process receiving a copy of acommunication between a first and second process. In some embodiments,the intercepting may be performed via an agent executing on a localmachine. In other embodiments, the intercepting may be performed by asub-routine, function, daemon, service, or application having theability to intercept communications between a three-dimensionalapplication and a three-dimensional rendering engine, which may comprisehardware or software for processing three-dimensional graphics commands.In some embodiments, the function call intercepted may be a functioncall that causes the contents of a back buffer to be presented to adisplay device. In these embodiments, the function call can indicatethat drawing or rendering has completed and that no more changes will bemade to the contents prior to presentation. In some embodiments,intercepting the function call may further comprise intercepting a callrepresenting a particular drawing region, such as the SetViewPort methodof the DIRECT3D API referred to above. In such embodiments, the functioncall may have data strings, variables, or other identifiers designatinga rectangle or other region of a rendering target. The designation of aregion may indicate that a modification of three-dimensional graphicsand graphics primitives may imminently occur within the region, hereinreferred to as a dirty region.

In one embodiment, the intercepted function call is used to identify thedesignated drawing region (step 804). In some embodiments, theidentification may be performed by the same agent, function, or serviceresponsible for intercepting the function call, and may be performed byparsing or analyzing the intercepted function call for a designation ofa region within a frame. In some embodiments, the function call mayinclude values designating a corner of a region and other valuesdesignating a width and height of the region. These values may beextracted to determine the location and size of the region. In otherembodiments, the function call may include values that may be utilizedto determine the location, size, and shape of the region. In still otherembodiments, the function call may lack values that designate a region,but may be accompanied by other function calls that do designate aregion or regions. In some embodiments, a further determination may bemade as to whether a designated region is in the back buffer that willbe imminently presented to the display device. In such embodiments andresponsive to the determination, a region may be excluded from theidentified dirty regions if it is not within the back buffer. Forexample, a region designated by a function call may be used as a rendertarget for textures and surfaces that are used to render the backbuffer, but may themselves not be part of the back buffer. In such acase, the region may be excluded from the identified dirty regions sothat the identified dirty regions only apply directly to the back bufferrather than indirectly.

In one embodiment, a determination is made (step 806) that a change hasoccurred to at least a portion of the graphics commands associated withthe dirty region or regions in the time since a previous interceptedfunction call. In some embodiments, this determination may be made byparsing or analyzing the contents of the identified dirty region orregions for three-dimensional graphics commands. In other embodiments,the determination could be merely made responsive to a region beingidentified in a function call, such as the SetViewPort method referredto above, or any similar function call for designating a region of aframe for drawing or rendering changes.

In one embodiment, the changed three-dimensional graphics commandswithin the dirty region or regions are extracted (step 808). In someembodiments, this may be a subset of the graphics commands within theentire frame, some which have not been changed in the time since aprevious intercepted function call. In other embodiments, the entireframe may be changed, while in still other embodiments, no changes willoccur within the frame.

Illustrated in FIG. 10 is one embodiment of a method 900 for delayingthe presentation of a back buffer surface to, in some cases, improveresource utilization. In this method 900, an agent intercepts a firstcall to a drawing library (step 902). A first frame is then transmittedto a remote computing device (904), and a determination is made as towhether the remote computing rendered the first frame (step 906). Thetransmission of a return message in response to the first call is thendelayed (step 908) until a notification is received that the first framewas transmitted or rendered (step 9 10). Upon receiving this notice,permission is granted to return the first call to the requestingapplication (step 912).

Further referring to FIG. 10, and in more detail, in one embodiment allthe steps of the method 900 may be carried out by an agent executing ona computing machine. Still other embodiments include an agent executingon any one of a local computing machine, a remote computing machine, athird computing machine, within a distributed computing system, on anetwork, on an appliance, or on any machine able to execute the agent.Still other embodiments include carrying out the method 900 in adistributed computing environment or system, where the distributedcomputing environment can include any combination of client computingdevice(s), server(s), appliance(s), three-dimensional applications, ornetwork(s). In one embodiment, the method 900 can execute in adistributed computing environment including a local computing deviceexecuting the agent and an application, where the local computing devicecommunicates with and is connected to a remote computing device withinthe distributed computing environment. Still other embodiments include adrawing library on any computing machine able to execute the agent,where the drawing library can be any of the above-describedthree-dimensional graphics libraries such as for example the DirectX orOpenGL libraries.

In one embodiment, an agent executing on the local computing machineintercepts a first call to a drawing library to further requestinformation from the drawing library (step 902). The first call is insome embodiments generated by an application executing on a computingmachine, and in other embodiments, is a call generated by an applicationexecuting on a local computing machine or server. Some embodimentsinclude a first call that is a first present call requesting to exchangea currently displayed image with an image to be rendered from the firstframe. In this embodiment, a hook may be installed in the drawinglibrary to prevent calls from returning to the requesting application.This can be done to control any one of how many frames are rendered andwhen frames are rendered on the local computing machine. When, in thisembodiment, the application requests graphical data or information, orgraphics primitives from the drawing library, the drawing library canreturn a rendered image. Thus, the return call is a return messagegenerated by the drawing library that can include a rendered image.

Still other embodiments include a method 900 where the agent transmitsthe first frame to a remote computing machine (step 904). There existembodiments where the agent transmits a plurality of frames as opposedto a first frame. In one embodiment, the agent which intercepts thefirst call is the substantially same agent that transmits the firstframe to a remote computing machine or device, while in otherembodiments, the agent that intercepts the first call is a substantiallydifferent agent than the agent which transmits the first frame. Thefirst frame, in some embodiments, can comprise any combination ofthree-dimensional graphics commands, rendered images, renderedthree-dimensional images, graphics primitives, and graphics libraries.

The agent, in one embodiment, determines the remote computing machinedid not yet render the first frame (step 906). In one embodiment, adetermination as to whether the remote computing machine has renderedthe transmitted first frame can be made by identifying whether theremote computing machine transmitted a confirmation to the localcomputing machine indicating that the first frame was rendered at theremote computing machine. Other embodiments include determining whetherthe remote computing machine has rendered the transmitted first frame bylistening to the network stack, and hearing from the network stack thatthe network is not busy. Still other embodiments include sending testpackets over the network to determine whether the network is busy,failing to receive a confirmation test packet, and determining that thenetwork is busy therefore the first frame has not yet been fullyreceived by the remote computing device. A determination can be made, insome embodiments, by analyzing the load placed on a network in betweenthe local computing machine and the remote computing machine todetermine whether the load has exceeded or fallen short of apredetermined threshold. When the load has exceeded the predeterminedthreshold, a determination can be made that the network is busy; whilewhen the load has fallen short of the predetermined threshold, adetermination can be made that the network is not busy and therefore thefirst frame has likely finished transmitting to the remote computingmachine and therefore is likely rendered.

In one embodiment, the return call to the requesting application isdelayed by the agent (step 908). Other embodiments include preventing toreturn the first call to the requesting application when it isdetermined that any one of the agent, local computing machine, orapplication failed to receive a notification indicating that the firstframe was rendered. The agent can, in some embodiments, delay thepresentation of the drawing by preventing the drawing library or anexecutable module in the drawing library from returning the first callto the requesting application along with the rendered graphics. When, insome embodiments, a graphic is rendered by the drawing library, suchrendering can take place on a back buffer that stores off screengraphics. In embodiments where the agent delays or prevents a return ofthe first call to the requesting application, the agent can be furtherdelaying or preventing the display or transmission of the contents ofthe back buffer to the requesting application. Still other embodimentsinclude preventing the return of the first call and accompanyinggraphics for a predetermined period of time, where such a period of timecan be determined empirically, randomly, dynamically, or can be hardcoded into the agent by a user or application. In another embodiment,the return of the first call may occur when a notification is receivedthat indicates the first frame was rendered, and after a predeterminedperiod of time.

Some embodiments include an agent that receives a notification that thefirst frame was rendered on the remote computing machine (step 910), andresponsively permits the return of the first call to the requestingapplication (step 912). In one embodiment, the notification is generatedby the remote computing machine, while in other embodiments thenotification can be generated by any one of the network, an applicationexecuting on the local computing machine, or an application executing onthe remote computing machine.

In one embodiment, method 900 includes an agent that either is aremoting mechanism or that interfaces with a remoting mechanism used totransmit graphics to a remote computing machine. This method 900 can, insome embodiments, delay rendering frames or graphical data by renderingafter a determination is made that the previous frame was successfullytransferred to the remote computing device, or alternativelysuccessfully rendered at the remote computing device. In anotherembodiment, after the agent permits a return of the first call to therequesting application, the agent transmits a second frame generated bythe application. In this embodiment, the agent receives a notificationthat indicates the second frame was rendered, and intercepts a secondcall generated by the application requesting information from thedrawing library. The agent can then identify an indicator that thesecond frame was rendered and the agent can permit the second call toreturn to the requesting application. In one embodiment, the agent candecide not to block a present call issued by the application when theagent determines that a flag has been received indicating that theprevious frame was received or displayed at the remote computingmachine. Thus, in this embodiment, the present call may not be blockedand may return to the requesting application.

The present disclosure may be provided as one or more computer-readableprograms embodied on or in one or more articles of manufacture. Thearticle of manufacture may be a floppy disk, a hard disk, a compactdisc, a digital versatile disc, a flash memory card, a PROM, a RAM, aROM, a computer readable medium having instructions executable by aprocessor, or a magnetic tape. In general, the computer-readableprograms may be implemented in any programming language. Some examplesof languages that can be used include C, C++, C#, or JAVA. The softwareprograms may be stored on or in one or more articles of manufacture asobject code.

While various embodiments of the methods and systems have beendescribed, these embodiments are exemplary and in no way limit the scopeof the described methods or systems. Those having skill in the relevantart can effect changes to form and details of the described methods andsystems without departing from the broadest scope of the describedmethods and systems. Thus, the scope of the methods and systemsdescribed herein should not be limited by any of the exemplaryembodiments and should be defined in accordance with the accompanyclaims and their equivalents.

1. A method for computing a hash from a three dimensional data setloaded into a three dimensional resource, the three dimensional data sethaving a variable size, the method comprising: determining anapplication, executing in a distributed computing environment, locked athree dimensional resource; identifying a first three dimensional dataset loaded into the three dimensional resource by the application, thefirst three dimensional data set having a size determined in part by atransaction carried out by the application; detecting the applicationperformed an operation on the three dimensional resource; computing afirst hash on the loaded first set of three dimensional data, the firsthash computed using a hash function; and storing the first hash in arepository
 2. The method of claim 1, further comprising: identifying asecond three dimensional data set loaded into the three dimensionalresource by the application, the second three dimensional data sethaving a size different from the size of the first three dimensionaldata set; detecting the application performed an operation on the threedimensional resource; computing a second hash on the loaded second setof three dimensional data, the second hash computed using the hashfunction; and storing the second hash in a repository.
 3. The method ofclaim 1, further comprising eliminating a portion of the first threedimensional data set responsive to detecting the application performedan operation, the portion of the three dimensional data set resultingfrom a repetitious transaction.
 4. The method of claim 1, whereindetermining further comprises determining the application locked a threedimensional resource used by the application.
 5. The method of claim 1,wherein computing using the hash function further comprises computingusing an adler-32 hash function.
 6. The method of claim 1, wherein thethree dimensional resource is one of either of a vertex, index, shade,texture, and surface.
 7. The method of claim 1, wherein identifying thefirst three dimensional data set further comprises identifying a firstthree dimensional data set having a size determined in part by atransaction carried out by the application, the transaction comprisingat least copying three dimensional data into the three dimensionalresource.
 8. The method of claim 7, wherein the transaction furthercomprises an end event comprising at least unlocking the threedimensional resource.
 9. The method of claim 2, wherein identifying thesecond three dimensional data set further comprises identifying a secondthree dimensional data set having a size determined in part by atransaction carried out by the application, the transaction comprisingat least copying three dimensional data into the three dimensionalresource.
 10. The method of claim 9, wherein the transaction furthercomprises an end event comprising at least unlocking the threedimensional resource.
 11. The method of claim 1, wherein identifying thefirst three dimensional data set further comprises identifying a firstthree dimensional data set having a size determined in part by atransaction carried out by the application, the size representative inpart of a period of time spanning from a start event to an end event.12. The method of claim 11, wherein the start event further compriseslocking the three dimensional resource.
 13. The method of claim 11,wherein the end event further comprises unlocking the three dimensionalresource.
 14. The method of claim 2, wherein the identifying the secondthree dimensional data set further comprises identifying a second threedimensional data set having a size determined in part by a transactioncarried out by the application, the size representative in part of aperiod of time spanning from a start event to an end event.
 15. Themethod of claim 14, wherein the start event further comprises lockingthe three dimensional resource.
 16. The method of claim 14, wherein theend event further comprises unlocking the three dimensional resource.17. The method of claim 1, wherein storing the first hash in arepository further comprises storing the first hash in cache.
 18. Themethod of claim 2, wherein storing the second hash in a repositoryfurther comprises storing the second hash in cache.
 19. A system forcomputing a hash from a three dimensional data set loaded into a threedimensional resource, the three dimensional data set having a variablesize, the system comprising: a distributed computing environmentcomprising at least a computing device executing an application, a firstrepository, and a second repository; and an agent executing on thecomputing device to: determine the application locked a threedimensional resource, identify a first three dimensional data set loadedinto the three dimensional resource by the application, the first threedimensional data set having a size determined in part by a transactioncarried out by the application, detect the application performed anoperation on the three dimensional resource, compute a first hash on theloaded first set of three dimensional data, the first hash computedusing a hash function executing on the computing device, and store thefirst hash in the first repository.
 20. The system of claim 19, whereinthe agent further: identifies a second three dimensional data set loadedinto the three dimensional resource by the application, the second threedimensional data set having a size different from the size of the firstthree dimensional data set; detects the application performed anoperation on the three dimensional resource; computes a second hash onthe loaded second set of three dimensional data, the second hashcomputed using the hash function; and stores the second hash in thesecond repository.
 21. The system of claim 19, wherein the agent furthereliminates a portion of the first three dimensional data set responsiveto detecting the application performed an operation, the portion of thethree dimensional data set resulting from a repetitious transaction. 22.The system of claim 19, wherein the agent determines the applicationlocked a three dimensional resource used by the application.
 23. Thesystem of claim 19, wherein the agent computes the first hash and thesecond hash using an adler-32 hash function.
 24. The system of claim 19,wherein the three dimensional resource is one of either of a vertex,index, shade, texture, and surface.
 25. The system of claim 19, whereinthe agent identifies the first three dimensional data set having a sizedetermined in part by a transaction comprising at least copying threedimensional data into the three dimensional resource.
 26. The system ofclaim 25, wherein the transaction further comprises an end eventcomprising at least unlocking the three dimensional resource.
 27. Thesystem of claim 20, wherein the agent identifies the second threedimensional data set having a size determined in part by a transactioncomprising at least copying three dimensional data into the threedimensional resource.
 28. The system of claim 27, wherein thetransaction further comprises an end event comprising at least unlockingthe three dimensional resource.
 29. The system of claim 19, wherein theagent identifies the first three dimensional data set having a sizerepresentative in part of a period of time spanning from a start eventto an end event.
 30. The system of claim 29, wherein the start eventfurther comprises locking the three dimensional resource, and the endevent further comprises unlocking the three dimensional resource. 31.The system of claim 20, wherein the agent identifies the second threedimensional data set having a size representative in part of a period oftime spanning from a start event to an end event.
 32. The system ofclaim 31, wherein the start event further comprises locking the threedimensional resource, and the end event further comprises unlocking thethree dimensional resource.
 33. The system of claim 19, wherein thefirst repository is a cache.
 34. The system of claim 20, wherein thesecond repository is a cache.